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ABSTRACT 


This  paper  concerns  the  first  phase  of  a  project  to  design  a  set  of  survey 
instruments  to  obtain  feedback  on  user  satisfaction  with  OTIC  products  and 
services.  Relevant  formal  user  studies  are  summarized  and  current  in-house 
efforts  to  obtain  user  input  are  outlined.  0MB  constraints  on  survey  research 
are  reviewed,  as  are  research  methodologies  which  appear  appropriate  to  obtain 
user  feedback.  A  survey  instrument  for  use  with  demand  bibliographies  is 
proposed.  Four  additional  procedures  for  product/service  evaluation  are 
recommended  for  the  next  phase  of  this  project. 
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INTRODUCTION 


One  of  the  responsibilites  of  the  Office  of  User  Services  is  to  recommend 
product  and  service  changes  and  additions.  In  order  to  suggest  changes  or 
additions,  it  is  necessary  to  first  determine  how  well  the  present  line  is 
serving  the  Defense  Technical  Information  Center's  (DTIC)  users.  That 
determination  can  only  be  made  by  the  users  themselves.  Therefore,  the  agency 
must  have  mechanisms  to  survey  its  users  to  obtain  feedback. 

Surveying  users  can  be  beneficial  for  an  agency,  beneficial  both  in  terms 
of  the  process  itself  and  in  terms  of  the  information  obtained  from  the  process. 
The  process  of  developing  a  collection  instrument  is  in  itself  beneficial.  It 
forces  tlie  researcher  and  the  persons  providing  the  product  or  service  to  stop 
and  analyse  what  they  do  and  how  they  go  about  doing  it.  Wide  staff 
participation  in  the  process  of  developing  the  feedback  mechanism  means  that 
more  people  will  learn  from  the  process  and  will  identify  with  the  findings  of 
the  survey. 

Obtaining  user  feedback  tends  to  have  a  positive  psychological  effect  on 
both  staff  and  users.  Staff  should  see  this  mechanism  as  a  means  to  obtain  an 
outside  credible  assessment  of  the  value  of  their  work.  The  work  should,  as  a 
result,  become  a  less  impersonal  process.  Further,  it  says  to  the  user  that  the 
agency  is  interested  in  providing  good  service  and  values  his  or  her  input. 
Contributing  such  feedback  gives  the  user  a  sense  of  having  an  active  role  in 
providing  for  his  or  her  own  information  needs. 

The  actual  results  of  the  survey  should  be  beneficial  to  agency  planners. 
From  such  results,  they  can  learn  more  about  tlie  characteristics  of  users, 
characteristics  of  the  products  and  services  they  want,  which  aspects  of  the 
existing  system  are  satisfactory  and  which  are  not,  the  need  for  user  education 


services  and  how  the  agency  is  perceived  in  relation  to  comparable  agencies. 
Having  assessed  user  wants,  agency  personnel  have  evidence  to  show  the  need  for 
actions  which  may  be  disruptive  to  the  organizational  climate.  Such  evidence 
from  consumers  should  facilitate  acceptance  of  the  changes. 

Finally,  survey  results  can  be  very  useful  to  management  in  an  era  of 
increased  demand  that  agencies  justify  their  expenditures.  At  one  time  it  was 
acceptable  for  agencies  to  describe  their  activities  to  their  oversight 
agencies  soley  in  terras  of  output  measures  such  as  bibliographies  shipped  or 
even  of  input  measures  such  as  so  many  staff  hours  worked.  This  failure  to 
justify  products  and  services  in  terms  of  benefits  to  users  is  no  longer 
acceptable.  Agencies  are  now  expected  to  be  able  to  provide  some  evidence  of 
an  impact  such  as  research  hours  saved  because  of  knowledge  of  prior  work. 
Similarly,  one  can  argue  more  effectively  that  increased  funding  is  justified 
if  one  can  show  systematic  efforts  to  use  funding  in  a  cost-effective  way. 

There  is,  therefore,  both  a  formal  requirement  and  a  substantive  need  for 
DTIC  to  have  a  set  of  mechanisms  to  obtain  on-going  user  input  as  to  the 
agency's  products  and  services.  Both  the  process  of  surveying  users  and  the 
results  of  the  survey  are  beneficial  because  they  help  to  rationalize  the  work 
process,  because  they  have  a  positive  psychological  effect  on  staff  and  users 
and  because  they  contribute  to  management's  ability  to  show  that  public  funds 
have  been  used  effectively. 

There  are  important  limitations  on  what  user  feedback  can  contribute. 
Surveys  of  user  populations  do  not  give  direct  information  about  non-users,  who 
might  in  fact  have  as  much  or  more  than  users  to  tell  the  agency  about  the 
value  of  its  products.  Another  important  limitation  is  the  fact  that  user 
feedback  will  provide  information  only  about  conscious  needs  the  user  believes 
the  agency  can  satisfy.  The  user  may,  in  fact,  have  important  needs  of  which 


he  or  she  is  unaware.  Additionally,  the  user  may  have  conscious  needs  which  etre 
left  unexpressed  because  he  or  she,  correctly  or  not,  believes  the  agency  can  not 
meet  those  needs.  Further,  the  information  provided  by  users  will  not  meet 
stringent  scientific  criteria  for  comparability.  The  data  themselves  will  be 
value  judgements  and  are  ordinal  measurements  at  best. 

Even  if  the  data  on  user  perceptions  is  accurate,  it  is  unlikely  to  be 
accurate  for  a  long  period  of  time.  User  expectations  are  an  everchanginq 
product  of  awareness  oF  needs  and  assessment  of  the  resources  available  to  meet 
needs.  The  changes  in  expectations  may  be  slow  but  they  will  occur.  Therefore, 
efforts  to  obtain  user  feedback  should  not  be  one-time  exercises. 

It  is  well  to  remember  that  the  user's  subjective  assessment  will  be  based 
on  a  number  of  factors,  including: 

1.  The  range  of  information  sources  available. 

2.  How  the  information  will  be  used. 

3.  The  individual  characteristics  of  the  user — motivation,  professional 
orientation,  experience  in  using  information  systems,  etc. 

4.  The  economic,  political  and  social  milieu  of  the  user. 

5.  Results  of  having  used  the  information . ^ 

Clearly  user  assessments  will  be  based  on  factors  not  entirely  under  the  control 
of  an  agency.  For  example,  the  assessment  of  a  demand  bibliography  will  be  based 
not  simply  on  the  search  results  per  se,  but  also  on  the  contents  of  the  data  base 
and  the  results  obtained  from  using  the  primary  literature  referenced.^  It  may, 
in  fact,  be  that  users  will  want  a  level  of  service  that  is  technically  or 
economically  impossible.  Further,  the  user,  feeling  these  needs  acutely,  may  be 
"sometimes  unable  to  understand  why  the  secondary  services  producer  cannot  always 
come  up  with  the  proverbial  goods''.'^ 

Information  on  the  degree  of  user  satisfaction  will  have  no  meaning  in  and  of 


itself.  Such  results  can  have  meaning  only  as  compared  to  the  objectives  of  the 
organization.  If  a  survey  of  users  Indicates  that  75%  of  them  are  satisfied  with 
the  results  of  a  search,  that  result  will  be  good  or  bad  depending  on  whether  the 
objective  was  50%  satisfaction  or  90%  satisfaction. 

The  fact  that  funds  are  expended  for  user  surveys  Implies  to  both  staff  and 
respondents  that  there  is  willingness  to  make  changes  that  will  be  of  tangible 
benefit  to  the  users.  However,  the  results  of  surveys  will  not  of  themselves  mean 
an  improvement  in  products  or  services.  Changes  come  about  because  management 
decides  that  results  are  not  consistent  with  objectives  and  has  the  resources  and 
committment  to  implement  a  possibly  very  difficult  change  process. 

In  summary  then,  the  development  of  survey  instruments  and  the  information 
which  is  gained  from  surveys  should  prove  highly  beneficial  to  the  agency.  The 
process  itself  will  promote  rational  analysis  of  OTIC  work  operations.  The 
results  of  surveys  will  aid  product  planners.  User  evaluations  should  promote  job 
satisfaction  because  DTIC  workers  will  have  outside  assessments  of  their  work. 
Further,  the  evaluations  will  signal  the  user  that  the  agency  is  concerned  with 
meeting  his  or  her  needs.  Of  no  small  importance  is  the  fact  that  user  feedback 
is  useful  both  in  program  advocacy  and  in  the  process  of  providing 
account abl 11 ty. 


ilils  project  has  been  divided  into  two  phases.  The  object  of  the  overall 
project  is  to  develop  a  comprehensive  set  of  feedback  mechanisms  to  measure  the 
degree  of  user  satisfaction  with  the  various  DTIC  products  and  services.  The 
specifics  of  the  full  set  of  feedback  mechanisms  will  be  considered  during  the 
second  phase  of  the  project.  The  steps  which  were  taken  during  the  first  phase 


sununarlzed  as  follows: 


1.  Review  of  previous  studies  which  could  be  expected  to  shed 
light  on  the  needs  of  OTIC  users.  These  studies  were  reviewed 
both  in  terms  of  content  with  respect  to  user  needs  and  in 
terms  of  suggestions  for  research  methodology  appropriate  for 
this  project. 

2.  Identification  of  the  various  ways,  both  formal  and 
informal,  that  DTIC  personnel  are  currently  using  to 
assess  user  perceptions  of  the  agency's  products  and 
services. 

3.  Assessment  of  the  ways  in  which  formal  constraints 
imposed  by  the  Office  of  Management  and  Budget  (0MB) 
could  affect  the  agency's  efforts  to  obtain  feedback. 

The  relevant  laws  and  administrative  determinations 
were  examined.  Several  other  information  handling 
agencies  were  contacted  to  learn  how  they  have  coped 
with  the  constraints  Imposed  on  survey  research. 

A.  Consideration  of  various  social  science  research 

methodologies.  Ones  that  appear  suitable  for  use  in 
measuring  DTIC  user  perceptions  are  briefly  described. 

5.  Presentation  of  a  possible  user  evaluation  form.  As  a 
first  step  in  the  developmental  process,  a  user  evaluation 
form  for  demand  bibliographies  is  offered  together  with 

a  plan  for  data  collection  and  analysis. 

6.  Finally,  recommendations  are  made  concerning  the  wider 
set  of  feedback  mechanisms  to  be  developed  during  the 
second  phase  of  the  project. 


EFFORTS  TO  MEASURE  USER  NEEDS  AND  PERCEPTIONS,  1965  TO  DATE 


Formal  Studies 


In  1963,  Alan  Rees  observed  that  the  information  retrieval  field  had  been 
plagued  for  many  years  by  busy  people  spending  large  sums  of  money  designing — or 
attempting  to  design — phantom  systems  for  nonexistent  people  in  hypothetical 
situations  with  unknown  needs. ^  Once  information  scientists  reali'^ed  that  one 
of  the  most  difficult  problems  confronting  their  field  was  the  determination  of 
the  true  nature  of  user  needs,  there  was  a  concerted  effort  to  study  the  user  in 
the  work  setting  Three  of  the  following  efforts — Auerbach,  1965;  North 
American  Aviation,  1966,  and  Auerbach,  1975 — examine  the  needs  of  the  DoD 
researcher  and  engineer  and  have  become  classics  in  the  user  study  genre. 

On  the  other  hand,  the  work  o..'  the  Committee  on  Information  Hang-Ups,  1975, 
was  an  attempt  by  working  information  professionals  to  describes  the  ways  that 
DTIC  could  best  serve  the  user.  The  1980  work  by  the  Institute  for  Defense 
Analysis  focused  on  the  ways  that  classification  of  part  of  the  data  base 
affected  service  to  users.  Finally,  the  Proceedings  of  the  DoD  Technical 
Information  Conference,  1981,  reflect  the  views  of  a  group  of  managers  concerned 
first  with  the  DoD-wide  information  program,  and  secondly,  with  DTIC's 
contribution  to  that  program.  These  studies  differ  in  their  approaches  but  all 
contribute  to  the  knowledge  base  concerning  the  needs  and  perceptions  of  DTIC 
users . 


Auerbach  Corj-xiration,  DoD  User  Needs  Study,  Phase  I,  1965 
The  purpose  of  this  DoD  sponsored  study  was  to  develop  a  comprehensive 


picture  of  the  information  seeking  patterns  of  tlie  DoD  RDT&E  community,  a 


heterogeneous/  inter-disciplinary  population  performing  a  wide  variety  of 
tasks.®  Since  there  was  relatively  little  prior  knowledge  about  the 
characteristics  of  this  population,  a  random  rather  than  a  stratified  sample  was 
chosen.  A  simple  random  sample  of  4%  of  the  population  was  taken.  In-depth 
interviews  were  conducted  by  trained  interviewers  at  the  user's  site.^  The 
interviewers  used  a  semi-structured  guide  intended  primarily  to  remind  them  of 
the  intent  of  the  questions.® 

The  study  was  deliberately  designed  to  avoid  the  measurement  of  opinion. 
That  is,  the  researchers  wanted  to  find  out  what  kinds  of  information  were 
actually  used  and  deliberately  avoided  questions  which  might  produce  comments  on 
the  kinds  of  information  service  the  user  might  like  to  have.  A  recently 
completed  task  was  isolated  and  the  characteristics  of  the  information  actually 
utilized  in  the  performance  of  the  task  were  ascertained.® 

This  technique  produced  a  large  body  of  data  concerning  the  characteristics 
of  the  population,  the  tasks  they  performed,  the  information  they  gathered  to 
perform  tasks,  and  their  use  of  information  services.^®  The  respondents 
were  found  to  be  well-educated,  mainly  civilian,  half  of  them  engineers,  and  a 
quarter  physical  scientists . ^ ^  This  community  was  found  to  be  involved  in  a 
wide  variety  of  tasks  with  the  greatest  emphasis  in  engineering.  The  major  task 

output  was  a  finding  or  a  recommendation  rather  than  a  decision,  and  the  task 

1 

results  were  usually  reported  in  writing. 

The  primary  emphasis  in  the  interviews  was  on  the  characteristics  of  the 
information  which  was  acquired  and  on  how  it  was  used.  They  found  that  the 
class  of  information  most  frequently  used  was  performance  characteristics  and 
specifications  (42%).  Approximately  a  third  of  the  respondents  reported  seeing 
only  one  item  of  the  available  material,  another  third  a  sampling  and  only  16% 
believed  that  they  had  had  access  to  all  the  available  material.  Seventeen 


percent  could  produce  no  answer  to  this  question  on  the  degree  of  recall  from 
the  relevant  knowledge  base.^^ 

When  asked  if  they  would  have  found  title  listings  or  cibstracts  useful  to 
read  before  selecting  materials,  16%  replied  that  such  aids  were  used  already, 
and  53%  replied  that  search  aids  would  be  useful.  However,  31%  believed  that 
such  aids  would  not  be  helpful. 

Colleagues,  personal  files,  and  local  departmental  sources  were  the  first 
sources  consulted  in  more  than  half  the  searches  for  information.  Oral 
communication  was  found  to  be  an  important  mode  of  acquisition  and  the  technical 
report  was  the  most  frequently  used  written  medium  (16%).  The  users  far  more 
frequently  wanted  a  specific  fact  or  a  detailed  analysis  such  as  a 
state-of-the-art  review  rather  than  a  bibliography  or  a  series  of  abstracts, 
labile  only  a  minority  of  the  respondents  reported  having  difficulty  finding 
information,  this  27%  minority  is  in  fact  a  sizeable  minority,  and  furthermore, 
the  user  niay  very  well  have  had  signficant  information  needs  of  which  he  or  she 
was  unaware. 

A  finding  with  very  direct  implications  for  DTIC  was  that  the  formal  DoD 
information  system  of  technical  libraries,  information  analysis  centers,  and  the 
then  Defense  Documentation  Center  (DDC)  was  not  widely  used.  One  reason  for 
this  underutilization,  was  found  to  be  a  lack  of  awareness  of  the  existence  of 
these  services.  The  researchers  suggested  that  this  lack  of  awareness  should  be 
attributed  to  a  lack  of  effective  publicity.  They  further  suggested  that 
another  possible  reason  for  low  usage  might  be  that  the  formal  system  did  not 
have  the  convenience,  responsiveness,  and  interactive  qualities  valued  by 
users. 

These  researchers  concluded  that  their  results  demonstrated  the  importance 
of  the  local  environment  and  confirmed  the  existence  of  an  informal  information 


system  in  the  DoD  RDT&E  community.  They  recommended  that  the  local  and  informal 
systems  be  strengthened  by  the  development  of  local  skills  directories  and  the 
improvement  of  the  personal  files  maintained  by  the  various  members  of  the 
technical  community.  To  improve  the  formal  information  system,  they  recommended 
a  piiblic  relations  program  to  enhance  the  awareness  of  the  end  user  of  services 
such  as  the  then  DDC  and  information  analysis  centers.  They  further  recommended 
the  development  of  conprehensive  training  programs  and  user's  guides  for 
intermediaries  such  as  local  librarians.  As  for  the  contents  of  the  formal  data 
bases  such  as  those  at  OTIC,  they  recommended  that  greater  emphasis  should  be 
placed  on  handling  engineering  data  such  as  performance  characteristics  and 
specifications. 


North  American  Aviation,  DoD  User  Needs  Study 
Phase  II,  1966 

Phase  I  of  this  study  concerned  the  information  needs  of  personnel  employed 
by  DoD.  Phase  II  of  the  study,  performed  by  North  American  Aviation  (NAA), 
investigated  the  information  needs  of  similar  personnel  in  the  defense  industry. 
Like  the  AAI  study,  the  NAA  work  is  based  on  a  large,  random  sample.  The 
sample,  in  this  second  case,  consisted  of  1,500  persons  from  83  organizations.^® 
Like  Phase  I,  this  phase  also  used  the  critical  incident  technique  to  focus 


response  on  specific  actions  as  opposed  to  user  perceptions.  The  interviewing 
technique  appears  to  have  been  much  the  same.^® 

Like  their  DoD  employed  counterparts,  contractor  respondents  found  that 


engineering  data  was  the  most  important  category  of  information,  specifically 
performance  and  characteristics  data  and  specifications.  There  was  slightly  more 
emphasis  by  contractors  (48%  of  needs  v.  42%  of  needs)  on  the  fields  of 
electronics  and  electrical  engineering,  and  aeronautics  and  space  technology. 
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Somewhat  fewer  contractor  personnel  found  the  local  work  environment  the  most 
important  first  source  for  information  (50%  for  contractors  v.  60%  for  DoD) ,  but 
a  greater  proportion  (46%  v.  39%)  had  their  information  needs  fully  satisfied  by 
local  sources  when  those  sources  were  the  first  ones  contacted.  When  using 
formally  organized  sources,  there  was  very  heavy  reliance  by  members  of  both 
groups  on  the  sources  such  as  departmental  files  in  the  local  environment.  The 
results  of  both  Phases  I  and  II  point  up  the  importance  of  information  analysis 
prior  to  distribution.  Both  communities  of  researchers  wanted  to  be  provided 
with  specific  answers  and  detailed  analyses. 

DoD  information  centers  and  services  were  found  to  be  underutilized  by  both 
DoD  and  industry  personnel,  with  only  about  45%  of  both  samples  using  those 
resources.  Approximately  one-fifth  of  DoD  workers  were  unaware  of  DDC's 
existence  as  were  almost  one-third  of  the  contractor  employees.  Practically  the 
same  proportion  of  DoD  personnel  as  contractor  personnel  were  unaware  of  the 
Technical  Abstract  Bulletin  (TAB),  i.e.,  40%  v.  43%.  More  contractor  personnel 
(42%  V.  27%)  had  problems  in  acquiring  information,  and  35%  attributed  problems 
to  security  restrictions  or  proprietary  limitations.  Approximately  a  fifth  of 
contractor  personnel  found  additional  task  information  after  the  task  was 
completed. NAA  further  learned  that  use  of  information  services  is 
interrelated,  e.g.,  users  of  TAB  are  likely  to  be  users  of  other  services. 

Based  on  the  results  of  their  study,  NAA  researchers  made  the  following 
recommendations  to  the  managers  of  the  DoD  information  programs: 

1.  Priority  of  effort  should  be  given  to  information  which 

is  in  the  development  phase  and  which  is  related  to  design 
and  performance  in  electronics  and  electrical  engineering 
and  in  aeronautics  and  space  technology. 


2.  Local  sources  of  information  should  be  strengthened  by 
(a)  making  more  readily  available  information  which  is 
informal  or  semiformal  in  composition,  (b)  tailoring  indexing, 
abstracting,  organization  and  analysis  of  information  prior 

to  its  distribution,  and  (c)  by  selective  and  automatic 
dissemination  of  these  tailored  indexes,  abstracts,  and  of 
organized  and  analyzed  information. 

3.  More  effort  should  be  addressed  to  satisfying  the  information 
needs  of  the  significant  users  of  DoD  information,  whcxn  they 
found  to  be  scientists  or  engineers  in  R&D,  holders  of 
advanced  degrees,  specialists  or  at  lower  management  levels, 
who  are  highly  paid,  that  is,  the  class  of  persons  who  appear 
to  be  the  greatest  users  of  information  centers  and  services 
and  the  ones  most  frustrated  in  their  efforts  to  use  these 
sources . 


Auerbach  Associates,  DDC  10  Year 
Requirements  and  Planning  Study,  1975-76. 

During  the  mid-1970's,  Auerbach  Associates,  Incorporated  (AAI),  were  again 
engaged  to  study  the  information  needs  of  the  DoD  community.  They  were 
specifically  contracted  to; 

1.  Identify  information  requirements  for  the  time  from  1978-1988. 

2.  Identify  end-user  problems  with  respect  to  services  provided 
by  DDC  and  other  sources. 

3.  Evaluate  DDC's  internally  established  long-range  objectives 
in  relation  to  the  findings  of  the  study. 

4.  Formulate  a  set  of  developmental  goals  for  1978-1988. 

5.  Describe  DDC's  role  in  the  information  community. 

To  study  user  requirements  and  problems,  AAI  conducted  a  survey  of  users 
and  jx3tential  users  using  mailed  questionnaires  and  a  highly  structured  schedule 
for  telephone  interviews. In  contrast  to  its  1965  study,  AAI  this  time 
used  a  stratified  sampling  plan  with  random  sampling  within  strata. 26 
schedule  was  designed  to  produce  data  on  the  user's  primary  sources  of 


information,  preferred  formats  and  media,  attitude  toward  user  charges,  awareness 
of  DDC  products  and  services,  perceptions  of  the  various  data  bases  and 
information  analysis  centers,  need  for  training,  current  problems  in  obtaining 
information  and  predicted  requirements  for  1978-1988.27 

AAI  also  surveyed  fourteen  federal  information  processing  agencies  selected 
for  their  relationship  to  DDC. 28  The  expert  review  technique  was  used  to 
develop  a  set  of  time-phased  assumptions  about  future  technological, 
organizational,  and  economic  conditions  as  they  could  impact  on  the  information 
system. 28  These  three  approaches  produced  a  considerable  body  of  data 
supporting  the  researchers*  conclusions  concerning  the  services  and  proposed 
plans  of  the  agency.  Three  over-arching  goals  were  recommended  for  DDC; 

1.  Maximize  the  potential  for  coordinating  the  RDT&E 
information  program  in  DoD. 

2.  Provide  new  and  improved  products  and  services. 

3.  Market  those  information  products  and  services. 

The  AAI  recommended  effort  which  is  of  special  interest  for  the  present  purpose 
is  Objective  II-2  which  called  for  the  agency  to  "establish  a  means  of  obtaining 
user  feedback  for  improvement  of  DDC  products  and  services . "2 "I 

Committee  on  Information  Hang-Ups,  1975 
About  the  same  time  as  the  AAI  study,  a  less  formal  effort  was  being  carried 
out.  In  1974,  the  Administrator  of  DDC  challenged  the  Committee  on  Information 
Hand-Ups  to  provide  substantive  input  into  that  agency's  ten-year  planning  study. 
From  the  membership  of  the  committee,  four  working  groups  were  assembled  to: 

1.  Evaluate  existing  services. 

2.  Identify  the  information  available  within  the  users 
organization. 


3.  Identify  the  relationships  in  the  DoD  information 
chain. 

4 .  Take  a  long-range  view  of  the  information  program  in 
OoD. 


In  contrast  to  the  report  produced  by  the  specialized  research  personnel 
employed  by  AAI,  this  report  was  produced  by  persons  in  the  DoD  information 
environment  who  continued  to  carry  full  workloads  within  their  own 
organizations. 

The  working  group  to  evaluate  the  various  DDC  services  developed 
questionnaires  and  distributed  them  to  members  of  the  Committee  on  Information 


Hang-Ups  and  to  certain  users  of  specific  services.  Based  on  responses  to  those 
questionnaires.  Subcommittee  I,  concluded  that: 


1.  DDC  users  were  not  as  well  informed  about  DDC  services 
as  they  believed  themselves  to  be. 

2.  DDC  users  believed  that  if  they  were  to  pay  for  services, 
they  were  entitled  to  services  of  high  quality. 

3.  That  the  Work  Unit  Information  System  (WUIS)  data  base 
was  valuable  and  deserved  DoD  insistence  on  much  more 
complete  input . 


4.  That  half  of  DDC's  efforts  should  be  directed  toward 
experimental  programs . 


After  studying  the  flow  of  information  in  DoD,  Subcommittee  II  concluded 
that  DDC  should  direct  the  flow  of  information  from  the  center  toward  the  user's 
library  rather  than  toward  the  end  user  himself  or  herself. Subcommittee 
III  recommended  that  the  center  expand  its  user  education  program  emphasizing 


specialized  services  such  as  the  Information  TVnalysis  Centers  (lAC).  The  members 
of  Subcommittee  IV  looked  within  their  own  organizations  and  identified  problem 
areas  related  to  lack  of  familiarity  with  DoD  and  DDC  services  and  the  need  for  a 
coordinated  information  program. 
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The  committee  as  a  whole  produced  a  number  of  specific  recommendations  for 
DDC,  recommendations  which  can  be  categorized  under  the  need  to; 

1.  Provide  continuing  education  for  DoD  information  personnel. 

2.  Promote  networking  among  federal  information  services. 

3 .  Support  innovative  technology . 

4.  Promote  standardization. 

5.  Lobby  for  the  free  flow  of  technical  information.^® 

Institute  for  Defense  Analysis,  1980 

In  late  1979,  the  Institute  for  Defense  Analysis  (IDA)  undertook  to  study 
ways  to  improve  access  to  DoD  information  through  the  DTIC  data  bases.  The  study 
was  to  consider,  in  part,  the  principal  objectives  involved  in  the  design  and 
operation  of  on-line  data  base  systems,  to  survey  a  sample  of  DROLS  users  to 
obtain  their  reactions  to  the  proposition  that  DROLS  become  an  unclassified 
system,  and  to  examine  the  utility  of  information  available  from  DROLS. 

The  IDA  researchers  surveyed  28  individuals  at  11  organizations.^® 

Interviews  were  set  up  by  telephoning  the  person  listed  as  the  DTIC  contact  in  a 
list  of  DROLS  terminal  sites.  This  person,  usually  an  information  source  person, 
was  sent  a  packet  of  materials  and  questionnaires.  After  the  contact  had  been 
given  time  to  examine  the  materials,  he  or  she  was  contacted  and  asked  to  line-up 
two  other  persons  for  interviews,  one  a  researcher  and  one  a  research  manager. 

Of  the  actual  interviews,  more  than  half  were  librarians  or  information 
specialists.-^  The  conclusions  which  could  be  drawn  from  the  study  were 
limited  by  the  fact  that  the  opinions  of  non-users  were  not  sought.  Further,  the 
hetrogeneous  character  of  the  organizations  employing  the  respondents  and  the 
small  size  of  the  sample  prompted  the  researchers  to  caution  against  too  much 
generalization  from  their  results.  Nevertheless,  they  felt  they  had  identified 


the  major  needs  perceived  by  most  DROLS  users.'*”  The  gist  of  most  of  the 
respondents  comments  was  that  DROLS  should  operate  more  like  the  commercial 
on-line  sy stems. 

From  their  review  of  the  pertinent  documentation  and  their  survey  of  users, 
the  IDA  researchers  identified  what  they  considered  a  discontinuity  in  the  DTIC 
mission.  That  is,  DTIC  is  committed  to  facilitate  the  transfer  of  information  in 
the  DoD/contractor  community  while  at  the  same  time,  it  must  also  control  and 
protect  classified  data.  The  first  commitment  implies  flexible  inexpensive 
access  and  compatibility  with  networks  and  other  retrieval  systems  while  the 
second  aspect  of  the  mission  requires  control,  expense,  complexity  and 
incompatibility  with  other  systems. 

While  they  concluded  that  real  advantages  would  be  obtained  by  the 
declassification  of  DROLS,  the  IDA  group  found  that  classified  users  were  so 
strongly  opposed  to  such  change  that  the  group  recommended  that  no  such  change  be 
made.^^ 

DoD  Technical  Information  Conference  for  R&D  Managers,  1981 
While  not  a  study  in  the  sense  of  the  items  previously  described,  the 
deliberations  and  conclusions  of  the  DoD  Technical  Information  Conference  for  R&D 
Managers  (1981  at  Ft.  McNair)  can  offer  useful  insight  into  the  proper  direction 
for  DTIC  efforts.  The  primary  purpose  of  the  conference  (sponsored  by  the  Deputy 
Under  Secretary  of  Defense  for  Research  and  Engineering)  was  to  bring  together  a 
large  cross-section  of  DoD  in-house  and  contractor  scientists,  engineers  and 
technical  managers  to  assist  in  planning  the  DoD  information  program.'*'* 

Although  it  was  not  described  as  such,  the  approach  could  be  likened  to  the 
research  technique  termed  the  "expert  panel  review."  The  conference  of  some  90 


persons  produced  a  number  of  recommendations  for  action  directly  related  to 
OTIC: 

1.  DTIC  should  be  designated  as  a  major  element  of  the 
defense  STIP. 

2.  DTIC  should  develop  a  data  base  locating  all  DoD 
sponsored  technical  reports. 

3.  DTIC  should  develop  a  system  to  evaluate  the  agency's 
technical  report  collection  including  evaluation 

of  subject  areas  for  balance  and  completeness  of  input. 

4.  DTIC's  ability  to  maintain  classified  data  bases 
should  be  retained  and  enhanced. 

5.  DTIC  should  work  with  the  military  agencies  to  improve 
access  by  contractors  to  specifications,  etc., 
required  to  respond  to  requests  for  proposals  and  bids.^^ 

King  Research,  1983 

In  1981,  King  Research,  Inc.  was  engaged  by  DTIC  to  survey  a  part  of  that 
agency's  users.  The  survey  was  conducted  and  the  data  has  been  at  least 
partially  analyzed.  The  final  report,  however,  was  not  available  at  the  time 
this  report  written. 

As  indicated  by  the  formal  studies  described  above,  a  variety  of  approaches 
can  be  taken  to  gain  understanding  about  what  DTIC  users  need  and  want.  It  has 
not  been  possible  to  implement  all  the  recommendations  in  these  studies. 

However,  many  suggestions  -  such  as  those  concerning  resource  sharing  and  dial-up 
access  -  have  been  implemented.  As  valuable  as  the  formal  studies  proved  to  be, 
they  were  discrete  efforts  and  DTIC  must  have  recurring  user  feedback 
particularly  for  certain  products  and  services.  DTIC's  efforts  to  obtain  that 
recurring  feedback  are  described  in  the  following  section. 
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In-House  Efforts  to  Obtain  User  Feedback 
On-Going  Operations 

The  various  directorates  have  established  mechanisms  which  as  part  of 
on-going  operations,  provide  user  feedback  in  varying  degrees  of  formality. 

DTIC  Form  141,  "Bibliography  Evaluation  Form"  (see  Appendix  B),  is  probably  the 
most  structured  mechanism.  The  Directorate  of  Data  Base  Services  (DTIC-T)  uses 
this  form  which  focuses  on  an  overall  evaluation  of  the  results  of  the  analyst's 
search.  It  is  additionally  intended  to  produce  an  estimate  of  the  time,  manpower 
and  money  saved  by  knowledge  of  prior  efforts.  These  forms,  which  have  been  in 
use  for  some  ten  years,  accompany  approximately  20%  of  the  bibliographies  sent  to 
DoD  users.  They  are  not  presently  sent  to  contractors  owing  to  0MB 
constraints.^®  Respondents  return  the  self-addressed  and  prepaid  forms  to  the 
Directorate  of  Planning  and  Management  (DTIC-M)  where  they  are  retained  for  two 
years. 

DTIC-M  analyzes  the  forms  particularly  in  terms  of  stated  cost  savings.'*'^ 
Examination  of  the  DTIC  Forms  141  in  the  DTIC-M  files  for  the  first  half  of  1982 
suggests  that  the  return  rate  is  quite  low  and  that  a  disproportionate  number  are 
returned  by  a  few  individuals.  For  example,  of  the  33  forms  in  the  file  of 
February,  1982,  18  were  returned  by  one  person.  To  the  extent  that  user 
complaints  via  the  Form  141  are  focused,  that  dissatisfaction  seems  to  center  on  a 
perceived  lack  of  relevance  of  the  searches  to  the  user's  needs. 

The  Current  Awareness  Bibliography  (CAB)  section  obtains  its  feedback 
informally  when  users  telephone  to  discuss  their  profiles.  No  record  of  this 
feedback  is  maintained.  In  addition,  the  annual  CAB  recertification  process 
acts  as  a  feedback  mechanism.'^® 

DTIC-T  uses  DTIC  Form  345,  "DROLS  Training  Evaluation,"  to  gain  user 
feedback  on  the  content,  format  and  presentation  of  online  class  training.  In 
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addition  to  the  two  evaluation  f  etas,  user  telephone  calls  and  the 


recertification  process,  the  manacrement  of  DTIC-T  also  recognizes  the  importance 
of  feedback  received  through  user  conferences  and  the  User  Services 
Off  ice. 

The  formal  feedback  mechanism  for  the  Directorate  of  Document  Services 
(DTIC-D)  is  the  position  identified  as  "Complaints  and  Inquiries  Processor." 

This  position  was  created  to  resolve  routine  operational  inquiries  regarding 
problems  such  as  mispulls  from  storage,  misshipments,  and  duplications  of  orders. 
A  monthly  summary  of  user  complaints  is  forwarded  to  DTIC-M.  Records  of  the 
specifics  of  the  complaints  are  destroyed  in  DTIC-D  after  six  months  as  it  is 
believed  that  six  months  is  the  maximum  time  period  for  corrective  action.  The 
former  incumbent  of  this  position  says  that  many  of  the  user  calls  are  not 
complaints  in  a  negative  sense  but  are  simply  requests  for  corrective  action. 

She  also  believes  that  much  of  the  user  feedback  coming  to  this  focal  point  fits 
into  categories  already  addressed  by  agency  procedures.^® 

The  project  manager  for  the  Automatic  Distribution  of  Documents  (ADD) 
program  tries  to  talk  to  each  of  the  program's  200  or  so  customers  at  least  once 
per  year.  He  encourages  them  to  review  their  profiles  and  check  their  shipments 
to  determine  whether  they  reflect  their  real  needs.  When  users  perceive 
problems,  those  problems  are  usually  related  to  the  finer  nuances  of  DTIC 
indexing. ^  ^ 

In  the  Directorate  of  ADP  Systems  (DTIC-S)  the  main  means  of  obtaining 
on-going  user  feedback  is  a  particular  telephone  number  which  users  have  been 
encouraged  to  call  when  they  need  immediate  action,  e.g.,  they  can  not  activate  a 
terminal  or  can  not  get  a  security  log  at  the  close  of  a  session.  The  emphasis 

here  is  on  solving  the  problem  at  hand  and  no  log  is  kept  of  the  problems 
52 


presented. 
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DTIC-S  has  no  feedback  forms  per  se ,  but  does  occasionally  participate  in 
special  studies.  For  example,  in  December,  1981,  the  directorate's  Management 
Support  Office  (DTIC-SM)  and  the  DROLS  Users  Council  asked  selected  users  to  time 
the  interval  between  their  transmission  and  return  of  the  cursor  to  the 
screen. In  another  instance,  in  March,  1982,  DTIC-SM  wrote  to  sites  which 
had  not  activated  during  1981  requesting  feedback  on  the  possible  reasons  for 
non-use  and  encouraging  them  to  contact  DTIC  if  they  were  experiencing  problems 
or  needed  further  information.^^ 

During  the  1960 's  and  much  of  the  early  1970 's,  DTIC  had  an  active  Customer 
Relations  Office.  Staff  members  were  available  for  on-site  consultations 
concerning  the  mission,  procedures,  and  services  of  DDC.^S  Additionally, 
this  office  provided  briefings  and  tours  to  visitors  to  the  center.  The 
identification  of  needed  modification  in  DDC  products  and  services  was  another 
responsibility  of  this  office. 

During  an  organizational  realignment  the  duties  of  the  Customer  Relations 
Office  were  disbursed  throughout  the  agency.  However,  the  need  for  a 
well-coordinated  user  liaison  program  was  apparent  (see  AAI,  1976)  and  in  1981 
the  function  was  recentralized  in  the  present  Office  of  User  Services  (DTIC-V). 
DTIC-V's  responsibilities  include  setting  up  a  liaison  program;  preparing  the 
DTIC  Digest,  user  guides,  and  promotional  materials;  operating  the  DTIC  regional 
service  facilities;  and  coordinating  with  and  supporting  user  groups.  This 
office  is  also  responsible  for  devising  means  to  obtain  user  feedback  and  for 
analyzing  that  feedback. Current  efforts  of  this  office  v/ill  be  noted  in 
connection  with  the  description  o^  feedback  mechanisms  of  longstanding  such  as 
user  conferences. 

From  time  to  time,  in-house  personnel  have  made  special  studies  not 
connected  with  on-going  operations.  ^or  example,  in  1979  as  an  intern 
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project,  Cotter,  Howes,  and  McCleery  distributed  a  questionnaire  to  attendees  at 


regional  user  conferences.  Based  on  the  responses  of  44  subjects  (56%  response 
rate)  they  concluded  that  many  users  lacked  knowledge  about  the  DDC  system  but 
that  the  shortcoming  was  not  "due  to  apathy  about  DDC  or  to  naivete  concerning 
on-line  systems  but  more  probably  reflected  the  agency's  failure  to  promote  its 
products  and  services."^® 


Contact  with  Organized  Groups 

Contact  with  professional  societies  and  user  groups  is  a  widely  accepted 
technique  for  promoting  an  organization’s  services  and  for  obtaining  customer 
feedback.  For  example,  DTIC  has  used  the  annual  conferences  of  the  American 
Library  Association  (ALA)  and  the  Special  Libraries  Association  (SLA)  as 
occasions  to  present  exhibit  booths.  Such  booths  provide  an  opportunity  to 
demonstrate  DTIC  services  to  a  nationwide  audience  and  obtain  feedback  from  a 
wide  range  of  users  and  potential  users. Similar  exhibits  were  also 
presented  at  the  last  two  national  conferences  of  the  American  Society  for 
Information  Science. Active  participation  in  the  Military  Librarian's 
Association  and  in  that  group's  annual  workshop  provides  another  particularly 
useful  means  to  encourage  the  DoD  information  community  to  see  the  agency  as  an 
approachable  institution.*^^ 

Another  organization  which  has  been  particularly  useful  to  and  supportive 
of  DTIC  in  its  efforts  to  communicate  with  users  is  the  Committee  on  Information 
Hang-Ups.  This  group  of  experienced,  Washington-based  information  specialists 
began  to  meet  in  1969.  They  gathered  together  to  discuss  solutions  to  problems 
encountered  in  use  of  tlie  aaency's  services.  After  conducting  a  user  survey  and 
producinq  a  report  titled  "Information  Hang-Ups:  Problems  Encountered  by  Users 
of  the  Technical  Information  Services  Offered  by  DDC  and  CFSTI",  the  group 
continued  to  meet  and  became  known  as  the  Committee  on  Information  Hang-Ups. 


The  group  has  remained  an  important  vehicle  for  communication  between  the  agency 
and  a  significant  segment  of  the  agency's  user  population.®^ 

Decisions  such  as  the  abolishment  of  regional  offices,  charges  for  document 
copies,  and  classification  cf  TAB  plus  the  success  of  the  Committee  on 
Information  Hang-Ups  in  Interacting  with  DDC  management  promoted  the  creation  of 
regional  user  groups.®^.  DDC  management  saw  such  groups  as  a  useful  means 
of  obtaininq  feedback  and  encouraged  them.  By  1971,  40  of  these  groups  had  been 
formed.®^ 

Many  local  groups  dissolved  but  by  1978  several  regional  groups  still 
survived.  In  that  year  DTIC  representatives  met  with  users  at  Northeastern, 
Southern,  and  Western  Regional  Conferences.  At  these  meetings,  users  provided 
the  agency  with  feedback  on  their  redesign  requirements,  on  the  need  for  lAC 
data  to  be  made  available  to  all  users  on-line,  reorganization  of  training 
manuals,  and  the  declassification  of  TAB.  Additionally,  they  provided  a 
critique  of  the  previous  annual  conference,  calling  for  better  meeting 
facilities,  better  preparation  by  workshop  leaders,  and  less  overall  formality. 
They  also  said  that  the  time  allotted  to  users  to  meet  without  DDC  personnel 
present  was  valuable  and  should  be  continued.®® 

The  following  year  also  saw  three  regional  user  conferences  —  in  New 
Mexico,  Florida,  and  in  Massachusetts.  That  year's  conferences  produced 
feedback  emphasizing  the  user's  requirements  with  respect  to  the  on-line  system 
files  and  formats.®®  Three  regional  meetings  were  held  in  1980,  again  in 
Florida  and  Massachusetts,  and  a  third  in  California.  That  year  users  expressed 
concern  regarding  interruptions  of  displays  and  prints,  delays  in  receipt  of 
batch  products,  and  the  progress  in  revision  of  the  training  manual.®^  In 
1981,  five  regional  conferences  were  held  —  in  California,  New  Mexico,  Florida, 
Massachusetts,  and  Virginia.  These  conferees  were  polled  on  their  reactions 


to  truncation  of  technical  report  abstracts  and  were  reminded  to  submit  DROLS 
trouble  reports  eind  to  contact  the  on-line  support  office  when  experiencing 
problems  with  NTIS  billing.®® 

The  recently  established  Office  of  User  Services,  DTIC-V,  views  the 
regional  conferences  as  a  planned  continuation  of  the  user  interaction  and 
feedback  established  at  the  annual  users  conferences.  DTIC-V  sees  these 
gatherings  as  an  "opportunity  for  interacting  with  as  many  users  as  possible  on 
progress  made  on  their  requirements,  on  new  policies  and  procedures,  to  react  to 
user  problems  and  questions  .  .  gain  significant  feedback  which 

is  "very  useful  to  DTIC  in  adjusting  its  range  of  services  and  developmental 
plans.  "■^0 

Six  of  these  regional  user  conferences  were  held  in  1982.  At  these 
meetings  the  users  reported  to  DTIC  their  concerns  regarding  cancellation  of  the 
Program  Planning  Data  Base,  slowness  in  limited  document  release,  input 
procedures  for  the  Shared  Bibliographic  Input  Network  (SBIN),  technical  report 
request  turnaround  time  and  on-line  response  time."^^ 

DTIC  users  have  organized  not  only  on  a  geographic  basis  but  also  around 
areas  of  interest,  i.e.,  the  on-line  retrieval  system  cind  SBIN.  At  the  1978 
annual  conference  DDC  suggested  that  the  on-line  users  organize  themselves  to 
voice  user  needs  to  the  agency. Volunteers  gathered  to  form  a  steering 
committee  and  agreed  to  begin  at  once  to  review  the  redesign  functional 
description,  review  and  consolidate  the  user  requirements  developed  at  regional 
meetings  and  urge  DDC  to  return  the  on-line  system  to  full-day  operation. 

These  volunteers  kept  in  contact  and  at  the  7th  Annual  DROLS  Users 
Conference  in  1980  a  group  of  government  and  contractor  users  was  officially 
organized,  to  be  known  as  the  DROLS  Users  Group.  The  group  elected  an  eleven 
member  council  to  represent  the  interests  of  the  military  services,  other  DoD 


agencies  and  the  contractor  community.  The  council  was  to  be  the  channel  for 
the  larger  group  to  make  recommendations  to  DTIC  concerning  matters  such  as 
search  capabilities,  telecommunications,  security  hardware  and  document 
delivery. The  council  has  been  active,  keeping  DTIC  aware  of  problem  areas 
such  as  incomplete  input  into  data  bases,  slow  response  time  and  misdirected 
mail.^^ 

The  participants  in  SBIN  have  also  chosen  to  organize  a  user  group.  This 
group,  the  Resource  Sharing  Advisory  Group  (RSAG),  was  established  with  DTIC 
encouragement  to  recommend  to  agency  management  mutually  beneficial  resource 
sharing  initiatives.^^ 

The  first  DROLS  Conference,  held  in  1973,  initiated  a  particularly  useful 
dialog  between  DDC  DROLS  specialists  and  on-line  users.  The  conferences  gave 
users  an  opportunity  to  interact  with  and  learn  from  each  other,  to  receive 
training  from  agency  personnel,  and  increasingly  over  the  years,  to  provide  the 
agency  with  input  on  user  needs.  For  example,  the  1978  annual  conference  was 
described  as  "well-attended  with  users  participating  freely  and  providing 
meaningful  dialog. That  year  users  complained  that  some  computer 
operators  released  and  reactivated  them  unnecessarily  causing  loss  of  user  file 
information,  that  they  were  slow  in  reacting  to  console  messages  and  that  some 
operators  were  rude.  On  the  other  hand,  the  operators  complained  that  some 
users  were  very  demanding  and  failed  to  follow  proper  procedures.^®  Other 
users'  concerns  related  to  updating  of  training  manuals  and  the  need  for  a 
directory  of  persons  having  expertise  in  particular  areas. Shared 
Bibliographic  Input  Experiment  (SBIE)  participants  that  year  expressed  a  need 
for  a  data  element  to  identify  site  entries  and  holdings  in  the  Technical 
Report  data  base.  The  site  representatives  wanted  this  data  element  so  that 
they  would  have  the  capability  to  create  online  catalogs  of  the  various  sites' 


holdings.®®  Subsequent  conferences  have  produced  feedback  on,  for  example, 
users'  desires  for  more  convenience  and  on-line  help  in  searching,  greater 
standardization,  and  more  current  terminology  for  indexing.®^ 

Since  1981,  the  annual  users  conference  has  been  coordinated  by  DTIC-V  and 
beginning  that  year  the  scope  of  the  meetings  has  been  expanded  to  include  users 
other  than  DROLS  sites.  The  final  portion  of  the  conference  has,  of  late,  been 
given  over  to  a  "Conference  Wrap-Up/Question  Answer  Session"  in  which  DTIC 
managers  respond  directly  to  questions  and  comments  by  attendees. ®2  This 
format  has  been  very  well  received  by  the  participants.  Overall,  these  national 
conferences  have  been  very  successful,  and  in  1981,  the  director  of  the  Defense 
Logistics  Agency  (DLA)  described  the  annual  conferences  as  "DTIC's  best 
opportunity  to  review  with  its  prime  users  the  effectiveness  of  its  services  and 
to  develop  suggestions  for  improvements."®® 

The  in-house  efforts  described  in  this  section  are  serious  attempts  to 
obtain  accurate  user  feedback  on  the  value  of  DTIC  products  and  services.  In 
point  of  fact,  these  methods  probably  do  give  the  majority  of  DTIC  managers 
fairly  realistic  impressions  about  how  well  certain  products  and  services  are 
meeting  the  needs  of  assertive  DTIC  users.  However,  it  should  be  possible  for 
all  managers  to  have  trustworthy  information  about  how  well  they  are  serving  the 
entire  population  of  DTIC  users.  If  DTIC  managers  are  to  have  this  vital 
information,  then  the  agency  as  a  whole  must  systematize  efforts  to  obtain  user 


feedback 


OMB  CONSTRAINTS  ON  SURVEY  RESEARCH 

The  Office  of  Management  and  Budget  (OMB)  is  responsible  not  only  for 
developing  and  administering  the  federal  budget  but  also  for  monitoring  the 
management  practices  of  federal  agencies.  In  December,  1980,  President  Carter 
signed  a  largely  unpublicized  piece  of  legislation,  the  Paperwork  Reduction  Act 
(Public  Law  96-511),  which  considerably  enhanced  OMB's  power  to  control  other 
federal  agencies  by  controlling  the  flow  of  information  into  and  out  of  the 
federal  government.  Federal  agencies  were  generally  displeased  by  PL  96-511,  a 
fact  that  did  not  disturb  the  bill's  major  supporters.  The  bill's  sponsor. 
Congressman  Jack  Brooks,  remarked  that  "this  bill  has  made  a  lot  of  folks  real 
unhappy  and  1  think  that's  just  fine."®'^  The  man  credited  with  much  of  the 
behind  the  scenes  work  on  the  bill,  Jim  Tozzi  of  OMB,  was  asked  if  the  Paperwork 
Reduction  Act  meant  that  OMB  would  have  control  over  all  information  going  out 
of  the  government.  He  replied:  "Yes,  but  that's  not  the  important  part.  We 
will  also  control  all  the  information  coming  into  the  government."®^ 

Some  observers  believe  that  the  bill  might  be  made  meaningless  because 
Appropriation  Committees  might  not  appropriate  sufficient  funding  for 
enforcement.  Others  see  a  danger  in  uneven  enforcement,  a  contingency  which  is 
in  fact  accepted  by  the  act.®®  Whatever  the  substantive  merits  of  the  bill 
and  the  hazards  of  its  implementation,  it  is  in  fact  in  force  in  DoD  and  must  be 
complied  with  in  collecting  information  from  the  public. 

The  law  establishes  within  OMB  an  Office  of  Information  and  Regulatory 
Affairs  to  be  administered  by  a  Director,  empowered  to  develop  and  implement 
federal  information  policies  and  to  review  and  approve  information  collection 
requests.'’  The  act  states  in  part  that  its  purpose  is  to  minimize  the 
federal  paperwork  burden  for  individuals  and  businesses  and  to  minimize  the  cost 


to  the  government  of  collecting,  maintaining  and  using  information  and  to 
maximize  the  usefulness  of  the  information  collected  by  the  federal 
government.®®  The  act  specifically  includes  military  agencies.®^  It 
defines  the  term  "burden"  to  mean  the  "time,  effort  or  financial  resources 
expended  by  persons  to  provide  information  to  a  federal  agency."®®  The  term 
"collection  of  information"  means: 

.  .  .  the  obtaining  or  soliciting  of  facts  or  opinions  by  an 
agency  through  the  use  of  written  report  forms,  application 
forms,  schedules,  questionnaires,  reporting  or  recordkeeping 
requirements  or  other  similar  methods  calling  for  .  .  .  answers 
to  identical  questions  posed  to,  or  identical  reporting  or 
recordkeeping  requirements  imposed  on,  ten  or  more  persons, 
other  than  agencies,  instrumentalities,  or  employees  of  the 
United  States.®^ 

The  term  "person"  is  defined  to  include  all  types  of  businesses  and 
associations. ®2  DTIC's  capacity  to  query  its  users  in  the  contractor 
community  is  thus  very  explicitly  constrained. 

If  OTIC  is  then  to  obtain  formal,  continuing  feedback,  it  must  submit  a 
Standard  Form  83A,  "Clearance  Request  and  Notice  of  Action."  The  clearance 
request  should  be  submitted  as  part  of  the  DLA  annual  Information  Collection 
Budget  (ICB).  The  ICB  is  an  estimate  of  the  total  number  of  hours  required  of 
the  public  to  comply  with  requests  for  information.  DLA-CM  expects  to  call  next 
for  submissions  for  its  ICB  in  May,  1983.  0MB  hearings  on  the  ICB  and  the 
various  information  collection  requests  accompanying  it  are  held  in  July  and 
August. ®^ 

PL  96-511  states  that  0MB  has  60  days  after  receipt  of  a  request  (plus  30 
days  extension  with  notice)  to  approve  or  deny  approval  of  an  information 
collection  request.  If  the  requesting  agency  receives  no  notification  after  90 
days,  approval  may  be  inferred,  a  control  number  assigned  and  the  agency  may 
collect  the  information  for  not  more  than  one  year.®^ 


Several  information  handling  activities  were  contacted  to  learn  how  they 
were  obtaining  user  feedback  in  light  of  these  constraints.  Only  one,  the 
National  Criminal  Justice  Reference  Service,  reported  having  tried  emd  been 
successful  with  the  0MB  clearance  process.  It  took  them  approximately  three 
months  to  get  0MB  clearance  for  a  survey  instrument. It  appears  that 
other  agencies  tend  to  either  rely  entirely  on  some  of  the  less  structured  means 
to  obtain  user  feedback  or  continue  to  use  old  survey  instruments  but  not  send 
them  outside  the  federal  government.  For  example,  the  Capitol  Systems  Group  no 
longer  makes  any  direct  survey  efforts.  They  believe  the  problems  involved  in 
getting  clearance  are  too  great.  Instead  they  use  other  indirect  ways  such  as 
face-to-face  contact  at  exhibit  booths  at  national  meetings.  Additionally,  user 
initiated  phone  calls  may  be  used  by  Capitol  Systems  to  inquire  into  areas  where 
feedback  is  needed.  They  also  believe  in  the  value  of  holding  workshops  and 
demonstrations  at  the  less  obvious  conferences.  For  example,  they  might  appear 
at  a  meeting  of  the  American  Association  of  Small  Research  Companies  as  opposed 
to  ASIS.  They  maintain  that  these  indirect  methods  are  good  ways  to  get  ideas 
and  sound  impressions  but  acknowledge  that  these  means  do  not  provide  data  in  a 
form  that  is  amenable  to  analysis.^® 

The  National  Aeronautics  and  Space  Administration  (NASA)  Scientific  and 
Technical  Information  Facility  continues  to  use  evaluation  forms  for  its 
searches  done  both  in-house  and  at  remote  sites.  These  franked,  return 
addressed  forms  are  part  of  the  transmittal  package  and  are  returned  at  about  a 
15%  rate.^^ 

The  Department  of  Energy  (DoE)  Technical  Information  Center  relies  heavily 
on  person-to-person  contact  to  obtain  feedback.  Center  representatives  hold 
briefings  in  the  field,  attend  meetinqs  at  other  agencies  and  host  user 
meetings.  All  of  these  means,  unfortunately,  involve  travel.  They  also 


announce  proposed  product  changes,  such  as  division  of  a  journal  into  more 


narrowly  defined  areas,  and,  as  part  of  the  announcement,  call  for  user  comment. 
They  consider  their  annual  recertification  process  a  feedback  mechanism.  The 
recertification  letter  is  accompanied  by  fact  sheets  about  the  various  services 
and  these  fact  sheets  request  user  comments  on  the  services  described.  The 
Technical  Information  Center  believes  that  by  these  indirect  means  they  are 
obtaining  credible  information  on  how  well  their  services  and  products  are 
received. 

The  National  Technical  Information  Service  (NTIS)  Customer  Relations 
section  also  relies  heavily  on  person-to-person  contact.  This  contact  is 
achieved  mainly  by  attendance  at  meetings  of  organized  groups  such  as  ASIS  and 
SLA.  NTIS  has  recently  initiated  a  campaign  to  establish  a  nationwide  network 
of  contacts  with  the  special  library  community.  Ruth  Smith,  Chief,  Office  of 
Customer  Services,  travels  to  meetings  of  SLA  chapters  to  present  information  on 
NTIS.  At  the  meetings  she  encourages  group  discussions  in  which  librarians 
produce  straightforward  assessments  of  NTIS  services  and  positive  suggestions 
about  how  NTIS  can  better  serve  them.  Smith  believes  that  a  user  group  can  be 
especially  helpful  in  view  of  present  0MB  constraints  if  that  user  group  chooses 
to  conduct  research  surveys  in  areas  of  concern  to  NTIS.  Additionally,  the  fact 
that  there  exists  an  Office  of  Customer  Services  attached  to  NTIS  management 
signals  to  users  that  NTIS  is  an  approachable  agency  which  will  listen  to 
advice. 

Users  of  these  unstructured  methods  probably  are  obtaining  largely  accurate 
impressions  of  how  well  their  services  are  being  received.  However,  study  of 
the  guidance  on  how  to  request  clearance  from  0MB  using  the  Standard  Form  83A 
suggests  that  the  process  may  in  fact  be  less  formidable  than  it  would  at  first 
seem  to  be.  There  appear  to  be  no  0MB  requirements,  in  terms  of  design  of 


research  or  in  design  of  survey  instruments,  which  are  not  consistent  with 


standard  research  methodology.  Quite  possibly  the  most  burdensome  aspect  of  the 
clearance  process  would  be  the  time  delays  involved.  Since  the  0MB  guidelines 
appear  reasonable  in  terms  of  criteria  for  research  design  and  since  DTIC  needs 
to  have  the  capability  to  query  the  entire  range  of  its  users,  it  would  be 
worthwhile  for  the  agency  to  develop  survey  instruments  which  0MB  will  find 


acceptable. 


INFORMATION  MODEL 


To  choose  among  the  various  options  for  data  collection,  one  must  first  be 
aware  of  the  actual  reasons  for  doing  a  survey.  For  example,  is  the  feedback 
from  a  search  evaluation  to  be  used  to  formulate  indexing  policy  or  is  it  to 
gather  testimonials  to  use  in  budget  justifications?  Both  are  worthwhile 
purposes  but  presumably  the  former  would  require  more  rigorous  instrument  design 
and  sampling  technique. 

Certain  standard  social  science  research  methods  were  rejected  as 
unsuitable  for  the  present  purpose.  For  example,  experimental  design  was 
rejected  because  it  is  most  suited  to  measure  the  actual  effects  of  change  as 
opposed  to  perceptions  of  a  condition  such  as  slow  response  time  or  of  an 
entity  such  as  TAB.  Experimental  design  might  at  some  time  be  used  if  DTIC 
managers  wanted  to  measure  the  effects  of  a  new  product  or  service.  However, 
experimental  design  will  not  be  further  considered  in  this  report. 

Direct  observation  of  behavior  is  a  costly  method  which  is  particularly 
recommended  for  studying  perceptions  about  sensitive  or  controversial  matters. 
This  method  was  rejected  both  because  of  cost  and  because  perceptions  about 
information  services  are  relatively  impersonal.  The  diary  method  and  the 
critical  incident  method  were  also  rejected  because  they  are  costly  and  are  most 
suitable  for  measuring  the  specifics  of  individual  behavior.  The  approaches  to 
be  considered  here  are  all  forms  of  self-report.  They  vary  in  their  degree  of 
structure  and  in  their  mode  of  administration  but  all  are  ultimately  dependent 
on  the  participation  and  openness  of  the  respondent. 

Questionnaires 

Self-administered  mailed  questionnaires  are  inexpensive  to  send  out  and 
they  save  on  interviewer  fees  and  travel.  They  may  be  inexpensive  overall 


provided  design  and  follow-up  costs  are  contained.  Another  advantage  of  the 
self-administered  questionnaire  is  that  it  is  frequently  easier  to  achieve  a 
more  representative  sample.  At  least  one  has  the  opportunity  to  solicit 
response  from  a  representative  sample,  for  example,  to  mail  questionnaires  to 
randomly  selected  user  codes.  By  contrast,  the  element  of  self-selection  would 
enter  if  the  same  set  of  questions  were  distributed  at  a  user  conference. 
Additionally,  respondents  may  be  more  frank  when  filling  in  an  impersonal  form 
than  when  face  to  face  with  an  interviewer.  The  potential  problem  of  bias  based 
on  personal  interaction  is  avoided  by  the  self-administered  questionnaire. 
Similarly,  the  mailed  questionnaire  gives  t)ie  respondent  an  opportunity  to 
assemble  data  if  such  is  necessary  and  gives  him  or  her  time  to  produce 
considered  answers. 

While  the  mailed  questionnaire  method  can  be  inexpensive  compared  to  many 
other  methods,  there  are  serious  problems  associated  with  it.  The  bias 
introduced  by  a  low  response  rate  can  seriously  mar  conclusions  drawn  from  the 
data  because  persons  who  voluntarily  complete  a  questionnaire  frequently  have 
characteristics  di'^ferent  from  those  who  do  not.^®^ 

Generally  speaking,  the  mailed  questionnaire  should  not  be  used  unless  the 
questions  are  clear-cut  and  unless  a  high  response  rate  can  be  anticipated. 
Questionnaires  should  be  as  brief  as  possible,  asking  only  for  the  minimum 
information  required,  and  asking  only  questions  that  can  and  will  be  answered 
truthfully.  One  should  avoid  jargon  and  ambiguity.  If  concepts  are  complex, 
e.g,  relevance  v.  recall,  explanations  will  be  necessary  and  the  researcher 
should  be  aware  that  many  respondents  will  skip  a  lengthy  explanation  and  record 
an  inappropriate  answer.  Several  questions  should  not  be  disguised  under  one 
question  and  the  designer  should  avoid  asking  respondents  about  events  distant 
in  time.  Generally,  the  questionnaire  should  ask  about  typical  behavior  rather 
than  behavior  during  an  arbitarily  selected  time  period.  Question.,  should  be 


phrased  neutrally  so  as  to  protect  the  respondent's  self-esteem.  Either  closed 
or  open-ended  questions  may  be  appropriate  but  the  designer  should  be  aware  that 
subjects  are  less  likely  to  respond  to  open-ended  questions  and  that  while 
open-ended  questions  can  be  coded,  the  coding  process  will  add  to  the  cost  of 
analysis.  Similarly,  either  general  or  specific  questions  can  meet  the  need. 

For  example  "Were  the  results  of  your  last  search  relevant  to  your  needs?"  may 
be  adequate.  However,  "Did  you  obtain  and  read  any  of  the  items  identified  on 
your  last  search?"  adds  a  greater  degree  of  specificity. 

One  should  try  to  present  a  question  sequence  that  is  natural  for  the 
respondent  and  particularly  the  first  few  questions  should  seem  important  to  the 
respondent.  Open-ended  questions  should  be  at  the  end  to  avoid  tiring  the 
respondent  before  other  data  is  gained.  Visual  quality  affects  response  rate 
and  one  should  seek  the  strongest  possible  sponsorship  for  the  survey.  One 
should  emphasize  the  importance  of  the  study  or  evaluation  to  the  respondent's 
own  interests — e.g.,  explain  that  the  purpose  is  to  better  tailor  bibliographies 
to  the  requester's  needs.  If  it  seems  practicable  to  do  so,  one  should  offer  to 
make  known  to  respondents  the  results  of  the  survey,  perhaps  through  the  Digest 
or  announcement  at  an  upcoming  conference. 

Unless  a  high  response  rate  is  obtained,  researchers  should  follow-up  with 
a  sample  of  non-respondents  so  as  to  learn  if  there  are  signficant  differences 
between  respondents  and  non-respondents.  The  costs  of  follow-up  may,  however, 
be  great  enough  to  offset  the  intial  low  costs  of  the  mailed  questionnaire. 

King  and  Bryant  advise  that  "it  is  almost  always  best  to  reduce  the  overall 
sample  size  and  insure  high  response  rate  by  fol  low-up ^ 

Interviews 

Inaccessibility  of  respondents  is  a  problem  that  pervades  this  method.  The 
problem  may  vary  from  the  person  who  is  difficult  to  reach  on  the  telephone  to 


the  larger  problem  of  bringing  a  group  together.  One-to-one  interviews  may  be 
carried  out  in  person  or  by  telephone.  Telephone  interviewing  avoids  travel 
costs,  an  important  advantage.  The  absence  of  visual  contact  may  be  a  problem 
since  the  interviewer  has  no  non-verbal  clues  as  to  the  respondent's 
comprehension  of  the  question.  On  the  other  hand,  lack  of  visual  clues  may 
lessen  distortion  created  by  interaction  between  respondent  and  interviewer. 

The  structured  one-to-one  interview  is  actually  much  like  the  mailed 
questionnaire.  It  should  be  as  carefully  designed  and  the  interviewer  should  be 
passive.  Quantification  is  still  relatively  easy  because  of  the  structure  but 
there  is  considerable  danger  that  opinion  will  be  recorded  inaccurately  using 
this  method  because  of  forced  categorization,  especially  if  the  respondent's 
views  are  not  crystalized. 

The  face-to-face  interview  is  a  very  flexible  way  to  obtain  information  and 
open-ended  questions  are  advised.  However,  this  approach  requires  considerable 
interpersonal  skill  on  the  part  of  the  interviewer  who  may  need  to  probe  for 
details  or  attitudes  that  are  not  expressed  overtly.  He  or  she  can  clarify 
questions  and  thereby  gain  more  accurate  information.  Admittedly,  interaction 
between  respondent  and  interviewer  can  distort  the  results  but  distortion  can  be 
partially  controlled  by  training  the  interviewers  and  by  dividing  interviews 
among  several  interviewers.  If  the  personal  in-depth  interview  seems 

the  best  technique,  but  if  funding  is  limited,  the  technique  of  sequential 
interviewing  may  be  used.  Basically  this  approach  consists  of  continuing  to 
interview  only  until  repetitive  information  is  being  received.  This  approach  is 
useful  in  hypothesis  generation  or  pretesting  but  does  not  provide 
representative  information. ^ Further,  what  one  gains  in  any 
unstructured  interview  is  impressions  which  may  not  be  readily  amenable  to 
analysis  or  quantification. 


The  group  interview  technique  takes  advantage  of  interaction  among  users  (8 
to  10  is  common).  With  a  moderate  level  of  guidance  from  a  facilitator,  the 
subjects  are  encouraged  to  discuss  their  perceptions  of  the  topic  in  question. 

A  permissive  atmosphere  created  by  a  skillful  moderator  and  a  supportive  group 
can  stimulate  the  expression  of  ideas  and  perceptions  with  a  spontaneity  not 
readily  obtained  in  individual  interviews.  Ideas,  for  example  policy  changes 
that  are  under  consideration  by  managers,  can  be  suggested  by  the  facilitator 
and  the  group's  responses,  suggested  modifications,  etc.  can  be  recorded  and 
considered  by  agency  management  before  finalizing  policy. 

Group  interviews  are  useful  when  one  wants  information  about  attitudes, 
beliefs  or  commonly  held  values  but  not  about  an  individual's  motivations  or 
behavior.  The  group  interview  is  not  the  way  to  get  answers  to  questions 
requiring  precise  detail. 

Consideration  must  also  be  made  for  differences  in  status  or  expertise 
level  of  members  of  the  group  and  the  effects  such  differences  will  have  on 
independent  thinking  and  candor.  The  danger  of  having  conclusions  distorted  in 
this  way  may,  however,  be  partially  controlled  by  using  group  interviews  as 
preludes  to  completion  of  individual  questionnaires.^®^ 

Choice  of  Sample 

Random  sampling  technique  is  suitable  "whenever  large  populations  are 
present,  innumerable  instances  that  have  an  outward  semblance  of  homogeneity  of 
structure,  a  large  aggregation  of  instances  that  evidence  common 
characteristics, — with  these  situations,  sampling  is  appropriate."^®^ 

That  is,  the  sample  should  be  chosen  such  that  it  reflects  in  accurate 


proportion  all  the  relevant  characteristics  of  the  total  population. 


Randomization  would  be  an  appropriate  way  to  select  respondents  for  a 
mailed  questionnaire.  For  example,  one  might  send  evaluation  forms  along  with  a 
random  sample  of  bibliography  orders.  Similarly,  the  researcher  could  randomly 
select  users  for  a  brief,  semi- structured  telephone  interview  survey.  One 
could,  in  theory  at  least,  select  a  random  sample  of  users  for  person-to-person 
interviews  or  even  for  group  interviews.  However,  travel  cost  and  time 
constraints  make  such  randomization  impractical.  It  is  likely  that  personal 
interviewing  would  have  to  be  done  at  conferences  and  the  inherent  bias 
accepted.  Such  bias  knowingly  accepted  and  openly  acknowledged  would  probably 
not  be  sufficient  to  invalidate  conclusions. 


Summary  of  Information  Model 


The  choice  of  research  method  is  much  affected  by  factors  other  than  the 
inherent  advantages  or  disadvantages  of  a  method  itself.  For  example,  if 
skilled  facilitators  are  not  available,  one  would  hesitate  to  arrange  group 
interviews  involving  travel.  Limited  funds  may  force  the  use  of  structured 
telephone  survey  when  a  more  costly  personal  interview  approach  would  be  more 
fruitful.  Similarly,  a  hastily  designed  evaluation  form  might  be  used  without 
pre-testing  if  time  is  too  short  for  further  work  on  the  instrument.  The 
ultimate  selection  of  a  survey  instrument  is  then  a  matter  of  determining  needs, 
balancing  needs  against  the  technical  advantages  of  the  various  research  design 
options  and,  finally,  considering  the  technically  optimum  design  in  light  of  the 
realities  of  what  is  available  in  time,  money  and  human  resources.  Having 
compromised,  the  researcher  must  acknowledge  the  limits  in  his  or  her  work  and 
be  open  to  new  evidence  which  might  challenge  his  or  her  conclusions. 

Of  the  techniques  outlined  above,  the  mailed  self-administered  structured 


questionnaire  appears  to  be  the  most  promising  technique  for  use  in  seeking 


feedback  on  many  of  DTIC's  recurring  products  v:hich  are  shipped  in  considerable 
numbers  such  as  TAB.  A  semi-structured  questionnaire,  either  self-administered 
or  administered  by  telephone  interview,  might  prove  a  very  productive  way  to 
gain  information  on  perceptions  of  non-recurring  services  or  irregularly  issued 
products  such  as  DRIT.  The  less  structured  personal  interview  could  prove  a 
very  useful  approach  to  take  at  regional  conferences,  annual  conferences  or  at 
other  times  and  places  where  users  might  be  conveniently  gathered  together. 

DTIC  users  have  shown  themselves  to  be  interested  and  articulate  observers  of 
the  agency  and,  as  a  consequence,  the  group  interview  technique  might  be  a  very 
effective  way  to  get  their  input,  depending  on  the  availability  of  a  trained 


facilitator. 


PROPOSED  DEMAND  BIBLIOGRAPHY  EVALUATION  FORM 


The  Report  (Demand)  Bibliography  was  chosen  for  the  initial  survey  effort 
because  it  is  a  product  clearly  central  to  the  DTIC  mission.  As  a  consequence, 
the  need  for  feedback  should  be  clear  to  an  oversight  agency.  Similarly  because 
the  item  is  tailormade  for  a  particular  researcher,  it  is  apparent  that  only  the 
recipient  can  evaluate  the  product's  effectiveness.  DTIC  has  no  real  means  to 
estimate  success  with  this  product  on  an  individual  or  an  aggregate  level.  In 
addition,  DTIC's  demand  bibliographies  are  appropriate  for  study  using  simple 
random  sampling.  This  is  true  because  as  a  collectivity  they  constitute  a 
generally  homogeneous  mass  of  individual  units.  The  randomization  process  will 
give  DTIC  greater  assurance  that  the  agency  is  getting  representative  feedback. 

Justification  for  Format 

The  format  for  the  proposed  survey  instrument  is  adapted  from  forms  which 
have  been  tested  through  use  by  DTIC  and  by  NASA.  It  addresses  the  central 
questions  of  recall  and  relevance,  timeliness  of  response  and  impact  on  DoD 
research.  It  does  not  address  factors  such  as  perceived  ease  of  access  to  the 
system,  or  the  form  of  the  output  product.  These  are  questions  that  might 
reasonably  be  asked  but  which  are  not  considered  central  to  the  analyst's 
retrieval  effort.  Similarly  it  does  not  inquire  into  matters  such  as  novelty 
and  perception  of  backup  document  delivery  capability.  These  are  questions 
frequently  addressed  in  evaluation  studies  but  not  addressed  here  owing  to  the 
specialized  nature  of  the  DTIC  data  bases  and  collection. 

The  evaluation  form  (See  Appendix  A)  begins  with  an  attempt  to  determine 
elapsed  time  between  order  and  receipt  of  results.  Item  4  of  the  form  seeks  a 
statement  of  the  requester's  topic  in  an  effort  to  lea  n  how  broadly  or  how 
narrowly  and  how  clearly  or  how  vaguely  his  topic  was  defined  in  his  own  mind. 
Knowing  how  he  intended  to  apply  the  search  helps  to  further  home  in  on  the 


question  of  recall  versus  relevance  as  it  was  defined  in  the  requester's  mind. 
Item  6  asks  much  the  same  thing  in  a  slightly  different  way.  Items  5  and  6  may 
be  merged  or  one  of  them  eliminated  in  the  pre-test  process.  Item  7  provides  an 
opportunity  for  an  overall  evaluation  in  a  familiar  scale  form  that  provides  for 
a  modera  e  degree  of  differentiation.  Item  8  again  addresses  the  question  of 
relevance  versus  recall  with  a  third  option  to  say  that  the  search  really  went 
awry.  The  cpiestion  on  the  number  of  citations  (Item  9)  is  intended  to  give  some 
sense  of  whether  the  recipient  found  the  results  sufficient  to  his  purpose 
regardless  of  how  extensive  he  believes  the  extant  material  to  be,  whether  he 
still  feels  the  search  and/or  the  literature  insufficient  for  some  reason  or 
whether  he  feels  burdened  by  an  excess  of  material.  The  tenth  q[uestion  is  a 
straightforward  inquiry  into  the  timeliness  of  the  receipt  of  the  product.  The 
eleventh  item  is  intended  to  find  out  if  the  researcher  understands  that 
negative  results  suggest  (if  not  prove)  that  prior  work  has  not  been  done  in  DoD 
and  that  zero  hits  on  a  topic  are  not  necessarily  a  fault  in  the  retrieval 
system.  Questions  12-15  are  designed  to  get  an  estimate  of  the  impact  of  DTIC's 
efforts  on  DoD  research.  Item  16  is  a  very  rough  measure  of  the  evaluator's 
knowledge  of  how  the  DTIC  system  works.  The  remaining  questions  concern  user 
identification  and  will  be  used  to  categorize  replies  and  to  follow-up  if 
clarification  is  needed. 


Data  Collection 

Evaluation  data  will  be  solicited  by  mailing  evaluation  forms  with  randomly 
selected  bibliographies.  A  relatively  small  sample,  perhaps  3%  or  less,  should 
be  adequate  if  persistent  follow-up  procedures  are  maintained.  It  is 
recommended  that  if  a  completed  form  is  not  received  within  two  weeks,  then  the 


recipient  should  be  requested  by  telephone  to  complete  and  return  the  form.  If 


no  reply  is  received  within  two  weeks  of  the  telephone  contact,  then  a  second 
attempt  should  be  made.  If  a  second  call  is  not  effective,  the  effort  should  be 
terminated  and  the  reason  recorded.  Such  follow-up  methods  will  require 
non-trivial  outlay  of  staff  effort.  However,  taking  a  small  sample  should 
minimize  the  expenditure  of  staff  time.  At  any  rate,  examination  of  files  of 
completed  DTIC  Forms  141  suggests  that  the  self-selection  factor  is  strong 
enough  to  seriously  distort  the  results  yielded  by  the  present  approach. 
Assertive  follow-up  measures  are  the  only  means  to  control  the  effects  of 
self-selection. 


Data  Analysis 

There  appears  to  be  no  requirement  for  sophisticated  statistical  analysis 
of  an  evaluation  of  this  type.  Simple  summations  of  responses  and  tabulations 
completed  quarterly  and  cumulated  annually  should  be  sufficient  to  provide  a 
picture  of  how  the  Demand  Bibliographies  are  being  received  and  how  researchers 
estimate  their  impact  on  the  DoD  R&D  effort.  Certain  items,  i.e..  Items  5,  6, 

7,  8,  9,  10,  11,  13,  and  16  are  structured,  closed  questions  and  should  be  easy 
to  record  for  analysis.  An  additional  category  for  each  item  would  be  added  for 
a  non-response  to  that  particular  item.  Items  2  and  3  would  require  some 
calculation  on  the  data  analyist's  part  and  may  prove  not  informative  enough  to 
continue.  Item  4  will  probably  be  difficult  to  categorize  neatly  except  in 
terms  of  a  specific  statement  (e.g.,  PPBS  or  anthropometry)  versus  a  less 
well-defined  one  (e.g.,  small  arms  and  aircraft  accidents.)  Item  12  on  how  the 
search  will  be  used  should  provide  examples  of  how  DTIC  has  made  a  contribution 
to  the  DoD  effort.  Items  14  and  15  ask  the  same  information  as  the  present  DTIC 
Form  141  and  could  be  analyzed  by  DTIC-M  following  present  procedures. 


RECOMMENDATIONS 


DTIC  needs  to  begin  at  once  to  develop  a  coordinated,  comprehensive  system 
for  obtaining  user  feedback.  Such  a  system  would  be  highly  beneficial  to  the 
agency  and  to  users.  The  process  of  developing  specific  survey  instruments 
would  force  staff  members  to  think  more  objectively  about  how  well  their 
products  and  services  actually  serve  the  purposes  they  are  intended  to  serve. 
Staff  would  be  encouraged  to  consider,  step  by  step,  the  various  processes  they 
go  through  to  achieve  their  goals.  This  careful  consideration  of  work  processes 
would  enable  staff  members  to  see  better  ways  to  do  their  jobs. 

Systematic  ongoing  effort  to  obtain  feedback  would  signal  to  users  that 
DTIC  recognizes  the  dynamic  nature  of  information  needs  in  research  and 
development.  Further,  such  an  effort  would  tell  the  user  community  that  the 
agency  acknowledges  the  value  of  its  users’  expertise  in  assessing  those 
changing  needs.  If  users  recognize  that  they  have  been  given  a  positive  role  in 
DTIC's  decision-making  process,  they  will  then  take  a  more  direct  interest  in 
the  success  of  the  agency. 

Having  the  recommended  input  from  users  would  provide  agency  managers  with 
practical  planning  information.  Planners  would  be  advised  of  the  specific 
characteristics  users  want  in  products  and  services.  They  would  also  be  alerted 
to  areas  where  user  training  is  particularly  needed.  Similarly,  reviewing  the 
results  of  surveys  over  a  number  of  years  would  enable  planners  to  more 
accurately  project  long-range  trends  in  user  needs. 

A  user  survey  system,  such  as  the  one  recommended  here,  would  provide 
managers  with  persuasive  evidence  to  use  in  their  efforts  to  justify  funding  for 


ct'jency  programs.  The  existence  of  a  set  of  carefully  designed  user  surveys 
would  confirm  the  fact  that  agency  managers  are  not  concerned  just  with 
maintaining  the  organization  per  se.  It  would  demonstrate  that  managers  have 
their  attention  focused  on  mission  accomplishment,  that  is,  on  efficiently  and 
effectively  meeting  the  actual  information  needs  of  the  defense  scientific  and 
technical  community.  The  findings  of  the  survey  system  would  constitute 
evidence  from  outside  the  agency  that  users  find  DTIC  products  and  services 
valuable.  Such  evidence  of  the  agency's  positive  impact  on  defense  research  and 
development  would  suggest  that  an  expanded  and  enhanced  program  would  produce 
even  greater  benefits. 

It  is  recommended  that  DTIC  take  the  following  actions  to  develop  and 
implement  a  systematic,  coordinated  program  for  obtaining  user  feedback; 

1.  DTIC  should  develop  a  set  of  self-administered,  mostly  structured 
questionnaires  to  be  mailed  with  certain  products  shipped  in  considerable 
numbers  on  a  regular  basis,  e.g.,  demand  bibliographies  and  hard  copies  of 
documents.  These  product  shipments  should  be  susceptible  to  the 
randomization  process.  The  sample  should  be  large  enough  to  be 
statistically  defensible,  but  no  larger  in  order  to  minimize  costs. 
Aggressive  follow-up  should  be  maintained.  Personnel  costs  for 
development,  distribution,  follow-up,  recordkeeping,  data  analysis  and 
reporting  would  range  from  $5900  to  $7300  during  the  first  year.  Appendix 
C  shows  cost  estimates  for  different  categories  of  personnel  effort  to 
implement  these  recommendations. 

2.  DTIC  should  develop  a  self-administered  essentially  structured 
questionnaire  covering  a  number  of  non-recurring  services  and  recurring 
products  and  services  not  covered  by  individual  evaluation  forms,  e.g., 
DRIT.  These  questionnaires  should  be  sent  to  a  small  sample  of  randomly 
selected  user  codes  and  follow-up  measures  completed.  It  is  recommended 
that  this  be  done  annually.  Personnel  costs  for  the  first  year  are 
estimated  to  range  between  $2600  and  $3250  (see  Appendix  C). 

3.  DTIC  should  develop  a  set  of  loosely  structured  personal  interview 
schedules  to  be  followed  in  obtaining  feedback  from  attendees  at  user 
conferences.  These  interview  schedules  should  be  flexible  enough  to 
accommodate  changing  technological  and  organizational  conditions.  At  the 
same  time,  they  should  be  sufficiently  structured  to  make  longitudinal 
comparison  possible.  The  sample  of  interviewees  should  he  chosen  at  random 
from  conference  confirmations.  However,  the  element  of  self-selection  will 
be  inescapable  because  attendees  at  user  conferences  will  tend  to  be  the 
more  active  and  interested  users  of  DTIC  products  and  services. 


Nevertheless,  the  perceptions  of  such  users  do  matter  and  they  should  be 
measured  in  a  way  that  makes  year-to-year  comparison  possible.  Personnel 
costs  would  range  between  $1900  and  $2400  (see  Appendix  C). 

4.  OTIC  should  develop  a  method  to  analyze  user  conference  minutes  to 
identify  and  categorize  issues  in  such  a  way  that  trends  from  year-to-year 
can  be  noted  and  recorded  as  such.  Certain  of  the  conference  sessions  seem 
to  be  much  like  over-sized  group  interviews  and  perhaps  similar  group 
dynamics  are  at  work.  At  any  rate,  costs  would  be  limited  to  the  costs  of 
the  analyst's  time  and  0MB  clearance  would  not  be  necessary.  Costs  of  the 
analyst's  effort  should  range  between  $400  and  $500  (see  Appendix  C). 


Staff  members  who  produce  the  product  or  service  to  be  evaluated  should 
participate  in  all  stages  of  any  attempt  to  develop  a  feedback  mechanism.  This 
is  essential  to  gain  their  substantive  input  for  development  of  the  surveys  and 
to  gain  their  cooperation  in  carrying  through  the  project.  Plans  for  population 
selection,  data  collection,  and  analysis  must  be  constructed  so  as  to  be 
defensibe  if  questioned  by  0MB.  All  survey  instruments  should  be  pre-tested  in 
the  Washington,  D.C.,  area  with  DoD  subjects  and  nine  or  fewer  contractors.  The 
final  survey  instruments  should  incorporate  the  changes  suggested  by  the 
pre-testing.  The  final  package  must  be  ready  for  incorporation  in  the  DLA 
Information  Collection  Budget  in  May.  Results  of  all  surveys  should  be  made 
known  to  users  via  appropriate  announcement  media. 

DTIC  would  derive  substantial  benefits  from  implementation  of  the 
recommended  program.  Such  a  program  should  not,  however,  be  viewed  as  static. 

It  would  require  periodic  re-examination  to  take  account  of  changes  in  the 


agency  and  in  its  user  community. 


APPENDIX  A 


INSTRUCTIONS  TO  THE  USER:  This  form  is  enclosed  in  OTIC  Report  Bibliogrephies  and  Oata  Bank  Summary  Reports.  The  objective 
is  to  obtain  information  for  improvement  of  OTIC  Retrieval  Services  to  customers.  The  evaluation  should  be  completed  by  the  principal 
user  of  the  information.  Return  to  OTIC  within  five  working  days,  if  possible. 


2.  SEARCH  CONTROL  NUMBER: 


3.  DATE  REQUESTED: 


4.  DATE  RECEIVED: 


5.  OVER  ALL  RETRIEVAL  EVALUATION  (Check appropriate  block) 


□  EXTREMELY  VALUABLE 


□  VALUABLE 


□  SATISFACTORY 


□  UNSATISFACTORY 


6,  IF  RESULTS  WERE  NEGATIVE,  WAS  KNOWLEDGE  OF  THIS  FACT  BENEFICIAL? 


□  YES  □  NO 


BASIS  FOR  THE  ABOVE  EVALUATION  (Pteax  specify): 


7.  DID  YOU  CITE  ALL  POSSIBLE  IDENTIFICATION  OF  THE  DESIRED  INFORMATION? 


IF  ANSWER  IS  NO,  PLEASE  LIST  OTHER  IDENTIFYING  INFORMATION  S/>eC//K.^: 


□  YES  □  NO 


DO  YOU  DESIRE  ANOTHER  SEARCH  USING  THIS  INFORMATION? 


□  YES  D  NO 


8.  WAS  THE  SEARCH  PRODUCT  BENEFICIAL  TO  YOUR  RESEARCH  OR  DEVELOPMENT  EFFORT? 


□  YES  □  NO 


A.  IF  ANSWER  IS  YES,  WASTHE  SEARCH  BENEFICIAL  IN  THE  SAVINGS  OF  TIME? 

ESTIMATED  AMOUNT _ 

(e.g.,  MAN-HOURS  SAVED  IN  INFORMATION  SEARCH  1  IME,  MAN  HOURS  SAVED  IN  AVOIDANCE  OF  WORK  DUPLICATION) 


B.  IF  ANSWER  IS  YES,  WAS  THE  SEARCH  BENEFICIAL  IN  SAVINGS  OF  MONEY? 

ESTIMATED  AMOUNT  _ 

(e.g.,  DOLLAR  VALUE  OF  SAVINGS  IN  INFORMATION  SEARCH  TIME;  DOLLAR  VALUE  OF  WORK  NOT  DUPLICATED,  ETC.) 


9.  IS  THIS  YOUR  FIRST  REQUEST  FOR  A  DTIC  BIBLIOGRAPHY? 


D  YES  □  NO 


12.  REQUESTER’S  NAME  AND  TITLE: 


13.  TELEPHONE  NUMBER: 


DTIC  141 


(FRONT) 


PREVIOUS  EDITIONS  ARE  OBSOLETE 


APPEJTOIX  A 


DTIC  BIBLIOGRAPHY  EVALUATION 


DATE 


INSTRUCTIONS  TO  THE  USER:  This  form  is  enclosed  m  OTIC  Report  Bibliographies  end  Dete  Bank  Summary  Reports.  The  objective 
is  to  obtain  information  for  improvement  of  OTIC  Retrieve!  Services  to  customers.  The  evaluation  should  be  completed  by  the  principal 
user  of  the  information.  Return  to  DTIC  within  five  working  days,  if  possible. 


1.  SEARCH  SUBJECT: 

2.  SEARCH  CONTROL  NUMBER: 

3.  DATE  REQUESTED: 

4.  DATE  RECEIVED; 

5.  OVER  ALL  RETRIEVAL  EVALUATION  (Check  appropriate  block) 


□  EXTREMELY  VALUABLE 


a  VALUABLE 


D  SATISFACTORY 


D  UNSATISFACTORY 


6.  IF  RESULTS  WERE  NEGATIVE,  WAS  KNOWLEDGE  OF  THIS  FACT  BENEFICIAL? 


□  YES  □  NO 


BASIS  FOR  THE  ABOVE  EVALUATION  (Please  Specify). 


7.  DID  YOU  CITE  ALL  POSSIBLE  IDENTIFICATION  OF  THE  DESIRED  INFORMATION? 


□  YF. 


□  NO 


IF  ANSWER  IS  NO,  PLEASE  LIST  OTHER  IDENTIFYING  INFORMATION  (Please  Specify). 


DO  YOU  DESIRE  ANOTHER  SEARCH  USING  THIS  INFORMATION? 

□  YES  □  NO 

8.  WAS  THE  SEARCH  PRODUCT  BENEFICIAL  TO  YOUR  RESEARCH  OR  DEVELOPMENT  EFFORT? 

D  YES  □  NO 

A.  IF  ANSWER  IS  YES,  WAS  THE  SEARCH  BENEFICIAL  IN  THE  SAVINGS  OF  TIME? 

ESTIMATED  AMOUNT _ 

le.g.,  MAN-HOURS  SAVED  IN  INFORMATION  SEARCH  TIME,  MAN  HOURS  SAVED  IN  AVOIDANCE  OF  WORK  DUPLICATION! 


B.  IF  ANSWER  IS  YES,  WAS  THE  SEARCH  BENEFICIAL  IN  SAVINGS  OF  MONEY? 
ESTIMATED  AMOUNT 


le  g.,  DOLLAR  VALUE  OF  SAVINGS  IN  INFORMATION  SEARCH  TIME;  DOLLAR  VALUE  OF  WORK  NOT  DUPLICATED,  ETC.) 


9.  IS  THIS  YOUR  FIRST  REQUEST  FOR  A  DTIC  BIBLIOGRAPHY? 

□  YES  a  NO 

10  USER  ORGANIZATION 

11.  USER  CODE; 

12.  REQUESTER'S  NAME  AND  TITLE; 

13.  TELEPHONE  NUMBER: 

DTIC  141 


(FRONT) 


PREVIOUS  EDITIONS  ARE  OBSOLETE 


aiBLlOGllAi’HV  LVALUAllON 

_ _ _ APPEN5IXJB _ _ 
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ii  'ij  sLaiii  intoiaut.'un  tor  impu',  •rnl  ot  D  Hi'  ll  li.c..,!  S.-nia  f  ■  a-  lama  v.  t hr  i  .  jlm,liun  ihoulJ  liv  cumplot$dLy  the (u'uxiptl 
an.'  i  f  1,'it  lr.tjin:o!.'on.  Raturn  13  DTIC  lu'thin  Ino  iwrkiaj  tbys.  it,;  ■iih'o.  _ 


iEAliCM  COMHUL  NUMbtRl 


~  m-k.-  - 

A.  tDcNTinCA'IION 


/  r.Ai i.  Ill  ctJ: 


3  DATE  RECEIVED: 


Mf  M  ^uL  Jl  C  1 
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ljl»  t- AO  h  L  1  Ht-' jl*t  C  T  IVC  St  All  Cm  /  .I  ,'fniriit  sur\  ty)  lull  U.,IVL  M  Alien  uri  A  •jfLCIFIC  ClUESTION 


OR  TOPIC 


Oli't  H  t^priijyl 


U  UDALI  I  Y 


le  •  (AJU  ue  I  A  ALL  LVALLIA  I  ION  Cl  I  I  Hi  it  AltCM  lirj  In,  1  (H.Ll  .VI.  JC.  ilALl  1  n'hcch  ant) 

V  p  ■  L'jtf  UL  .  or  I  niLi; 

o  S 1  I  ‘  S  C 

V  _ ¥ _ ¥  . .  V. . 


...¥. _ 


OP  NO 

use 


Cu;  voo  HNL  me  SLAliCM  TO  ot:  L]  .1.  CUMPHCMEN'jl.t 

;  I  1.  f<;  iTHin  CCMriUMENGIVe  MOR  (fjT.fJalVt 


{.1  t.  IflTENSIVb 
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I  !  iT  1,1  .  v'Lib  iVi-f  it  fJtGATIVE,  WAS  KNC.VLfiOOii  Of  THIS  HACT  tlLNCHlCIAL? 


a  YES  □  NO 


C.  application 


I  ,„jv.  /-I. I  you  GOING  TO  USt  THIS  SEmHCHI 


;  '  V./.  .  i  ME  SEARCH  PRODUCT  BENEFICIAL  TO  YOUH  RESEARCH  OH  OtVCLOl'MCNT  CFCOFIT?  DYES 

j  ■*  ’ll  a:. C'l’.Cl'l  15  YES,  WA5THE  SEARCH  OuNEFICIAL  IN  THE  SAVINGS  OF  Tir^E? 

■  [  ^1  I.MATEOAMOUNT  _ 

j  I,  J  ,  hours  Saved  in  information  search  TI.ME.  man  holm  i  SAVED  IN  AVOIDANCE  OF  WORK  DUPLICATION) 

*  ^  If  ANCNCH  IS  YES,  WASTHE  SEARCH  BENEFICIAL  IN  SAVINOSOK  MONEY? 

ESTIMATED  AMOUNT  _ _ 


(i  9  ,  DOLl  AR  VALUE  OF  SAVINGS  IN  INFCMMATION  SEARCH  TIME;  DOLLAR  VALUE  OF  WORK  NOT  DUPLICATED,  ETCJ 


1  io  iS  This  YOUR  FIRST  REQUEST  FOR  A  DTIC  BIBLIOGRAPHY? 


APPENDIX  C 


COST  ESTIMATE  FOR  RECOMMENDATION  NUMBER  ONE 


DEMAND  BIBLIOGRAPHY  EVALUATION  FORM 


Developmental  costs 

Coordination  with  retrieval  analysts  GSll/32hrs  $38A 

Design  form  GSll/24hrs  288 

Pre-test  form  GSll/24hrs  288 

Design  coding  sheets  GSll/08hrs  96 

Set-up  filing  system  GSll/04hrs  48 

. total  =  1104 

Distribution,  recordkeeping,  and  follow-up 

Assign  random  numbers  to  bibliographies  GS09/03hrs  30 

Insert  forms  with  bibliographies  GS05/68hrs  442 

Distribution  notice  to  DTIC-V  GS05/34hrs  221 

File  distribution  notices  GS05/34hrs  221 

Match  distributions/returns  GS09/34hrs  340 

ID  subjects  for  follow-up  GS09/34hrs  340 

1st  follow-up  calls  (327  on  20%  rtn  rate)  GSll/55hrs  660 

File  1st  follow-up  returns  GS05/34hrs  221 

2nd  follow-up  calls  (111  on  66%  rtn  rate)  GSll/19hrs  228 

File  2nd  follow-up  returns  GS05/34hrs  221 

. total  =  2924 

Data  analysis  and  reporting 

Code  data  from  80%  return  GSll/109hrs  1308 

Analysis  GSll/  16hrs  192 

Write  report  GSll/  24hrs  288 

. total  =  1788 

COST  ESTIMATES  SUMMARIZED  LOW  -  HIGH 

Developmental  1100  -  1375 

Distribution,  recordkeeping,  follow-up  3000  -  3750 

Data  analysis  and  reporting  1800  -  2250 

.  total  =  5900  -  7375 


Estimate  is  based  on  the  number  of  bibliographies  shipped  in  1982: 

TR  9047  X  .03  sample  =  271 

WUIS  3009  X  .04  sample  =  120 

IR&D  451  X  .04  sample  =  18 


COST  ESTIMATE  FOR  RECOMt-lENDATION  NUMBER  TWO 


NON-RECURRING  PRODUCTS/SERVICES 

EVALUATION  FORM 

Developmental  costs 

Design/ coord inate  form 

Pre-test  form 

GSll/60hrs 

GSll/24hrs 

Distribution 

Assign  random  numbers 

Produce  labels/mailing 

GS09/02hrs 

GS05/03hrs 

Follow-up  procedures 

1st  follow-up  call  (84  on  20%  rtn  rate) 

2nd  follow-up  call  (40  users) 

Remail  15%  of  forms  (16) 

GSll/42hrs 

GSll/20hrs 

GS09/01hrs 

GS05/01hrs 

Analysis  and  reporting 

Receive  and  file  returns  (est.  89) 

Analysis  of  89  forms 

Write  report 

GS09/15hrs 

GSll/45hrs 

GSll/12hrs 

COST  ESTIMATES  SUMMARIZED 

Developmental 

Distribution  and  follow-up 

Analysis  and  reporting 

EstLiaate  is  based  on  the  number  of  OTIC  users  (3512)  as  of  November,  1982  and 
3/o  sample  of  105  user  codes. 


a 


PERSONAL  INTERVIEW  SCHEDULE 


Developmental  costs 

Design  schedule 

Pre-test  schedule 

Train  2nd  Interviewer 

GS09/40hrs 

GS09/24hrs 

GSll/08hrs 

GS09/08hrs 

Administration  of  schedule 

Select  subjects 

Call  prospective  subjects 

Confirmation  calls  to  subjects 

Interviews  (24) 

GSll/02hrs 

GSll/12hrs 

GSll/12hrs 

GS09/12hrs 

GSll/12hrs 

] 

] 

] 

1 

Analysis  and  reporting 

Consultation  (2  interviewers) 

Data  analysis 

Write  report 

GS09/02hrs 

GS09/02hrs 

GS09/12hrs 

GSll/12hrs 

GSll/24hrs 

] 

] 

2 

total  =  ; 

COST  ESTIMATES  SUMMARIZED 

Developmental  costs 

Administration  of  schedule 

Data  analysis  and  reporting 

LOW  -  H] 
800  -  1C 
575  -  ; 

600  -  / 

Estimate  is  based  on  275  attendees  at  the  1982  conference.  Interviews  with  24 
subjects  are  recommended. 


COST  ESTIMATE  FOR  RECOMMENDATION  NUMBER  FOUR 


ANALYSIS  OF  USER  CONFERENCE  MINUTES 


Developmental  costs 

Examine  minutes  and  categorze  issues 
Design  coding  sheet 
Write  report 


GS09/12hrs 

GS09/12hrs 

GS09/16hrs 


COST  ESTIMATE  SUMMARIZED 
Development  and  reporting 


$120 

120 

160 

.total  =  400 

LOW  -  HIGH 
400  -  500 
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